In statistics, the Holm–Bonferroni method [1] performs more than one hypothesis test simultaneously. It is named after Sture Holm and Carlo Emilio Bonferroni.
Contents |
Suppose there are k null hypotheses to be tested and the overall type 1 error rate (significance level) is α. Start by ordering the p-values and comparing the smallest p-value to α/k. If that p-value is less than α/k, then reject that hypothesis and start all over with the same α and test the remaining k − 1 hypothesis, i.e. order the k − 1 remaining p-values and compare the smallest one to α/(k − 1). Continue doing this until the hypothesis with the smallest p-value cannot be rejected. At that point, stop. None of the remaining hypotheses can be rejected.
Here is an example. Four null hypotheses are tested with α = 0.05. The four unadjusted p-values are 0.01, 0.03, 0.04, and 0.005. The smallest of these is 0.005. Since this is less than 0.05/4, null hypothesis four is rejected (meaning some alternative hypothesis likely explains the data). The next smallest p-value is 0.01, which is smaller than 0.05/3. So, null hypothesis one is also rejected. The next smallest p-value is 0.03. This is not smaller than 0.05/2, so you fail to reject this hypothesis (meaning you have not seen evidence to conclude an alternative hypothesis is preferable to the level of α = 0.05). As soon as that happens, you stop, and therefore, also fail to reject the remaining hypothesis that has a p-value of 0.04. Therefore, hypotheses one and four are rejected while hypotheses two and three are not rejected. Applying the approximate false discovery rate produces the same result without requiring ordering the p-values, then using different criteria for each test.
The Holm–Bonferroni method is an example of a closed test procedure.[2] As such, it controls the familywise error rate for all the k hypotheses at level α in the strong sense. Each intersection is tested using the simple Bonferroni test.
It is also possible to define a weighted version. Let p1,..., pk be the unadjusted p-values and let w1,..., wk be a set of corresponding positive weights that add to 1. Without loss of generality, assume the p-values and the weights are all ordered such that p1/w1 ≤ p2/w2 ≤ ... ≤ pk/wk. The adjusted p-value for the first hypothesis is q1 = min{1,p1/w1}. Inductively, define the adjusted p-value for hypothesis i by qi = min{1,max{qi−1,(wi + ... + wk)×pi/wi}}. A hypothesis is rejected at level α if and only if its adjusted p-value is less than α. In the earlier example using equal weights, the adjusted p-values are 0.03, 0.06, 0.06, and 0.02. This is another way to see that using α = 0.05, only hypotheses one and four are rejected by this procedure.